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WHAT IS CLAIMED IS: 

1. A control system for controlling a display device 
including a display unit displaying a motion picture, and a 
pointing unit pointing a position on the motion picture, said 

5 control system comprising: 

a recognizing unit recognizing an object in the motion 
picture on which the pointing manipulation is effected; and 

a processing unit executing a predetermined process 
related to the object recognized. 

10 

2. A control system according to claim 1, wherein said 
display device further includes a receiving unit receiving data , 

said receiving unit receives first data containing the 
motion picture, and second data related to the object in the 
15 motion picture contained in the first data, and 

said processing unit makes said display device output 
the second data related to the object recognized. 

3 . A control system according to claim 2 , wherein the second 
20 data contains an image related to the motion picture of the first 

data . 

4 . A control system according to claim 2 , wherein the second 
data is multiplexed with the first data and thus distributed. 

25 

5 . A control system according to claim 4 , wherein the second 
data is embedded in a margin of the first data. 
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6 . A control system according to claim 2 , wherein the motion 
picture of the first data contains an image related to a commercial 
article or a service becoming an advertisement target, and 
5 the second data is an advertisement of the article or the 

service . 


7. A control system according to claim 1, wherein said 
processing unit executes an application program related to the 

10 object recognized. 

8. A control system according to claim 1, wherein said 
processing unit executes a task related to the obj ect recognized. 


15 9. A control system according to claim 1, wherein said 

display device further includes a communication unit 
transmitting a command to other task executing device, and 

said processing unit, via said communication unit, 
commands said other task executing device to execute a task 

20 related to the object recognized. 


10. A control system according to claim 9, wherein said 
task executing device is an E-mail server delivering an E-mail, 
and the command is a request for transmitting E-mail data related 
25 to the object, and 

said processing unit receives the transmitted E-mail data 
via said communication unit. 
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11. A control system according to claim 1, wherein said 
display device further includes a communication unit 
transmitting information to an add-up server adding up pieces 

5 of information, and 

said processing unit, via said communication unit, 
notifies said add-up server of the information related to the 
recognized object, and makes said add-up server add up results 
of the recognition. 

10 

12. A control system according to claim 2, wherein the 
motion picture is formed of a plurality of frames, 

said receiving unit further receives definition 
information for defining an area of the object of each frame, 
15 and 

said recognizing unit collates a position on the motion 
picture receiving the pointing manipulation with the definition 
information with respect to the frame of the motion picture 
receiving the pointing manipulation. 

20 

13. A control system according to claim 12, further 
comprising a unit displaying the area in the motion picture with 
enhancement . 

25 14. A control system according to claim 2, wherein said 

receiving unit further receives information for describing a 
characteristic of the object, and 
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said processing unit generates an object corresponding 
to the motion picture on the basis of the motion picture and 
the information for describing the characteristic of the obj ect . 

5 15. A control system according to claim 1, wherein when 

the pointing manipulation occurs during the display of the motion 
picture contained in the first data, said processing unit, after 
an end of displaying the motion picture, commands said display 
device to output the second data related to the object recognized 

10 by the pointing manipulation. 

16. A control system according to claim 15, wherein said 
display device further includes a recording unit recording data , 
and 

15 said processing unit, when the object is recognized from 

the pointing manipulation during the display of the motion 
picture, commands said recording unit to record the second data 
related to the obj ect , and after the end of displaying the motion 
picture, commands said display device to output the second data 

20 recorded. 

17. A control system according to claim 1, wherein said 
processing unit, when the obj ect is recognized from the pointing 
manipulation during the display of the motion picture contained 

25 in the first data, commands said display device to effect a first 
changeover to an output of the second data related to the object. 
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18. A control system according to claim 17, wherein said 
processing unit commands said display device to executes a second 
changeover to displaying the motion picture after outputting 
the second data . 

5 

19. A control system according to claim 18, wherein said 
display device further includes a recording unit recording data, 
and 

said processing unit, after outputting the second data, 
10 issues a command of displaying the motion picture from a point 
of the first changeover recorded on said recording unit. 

20. A control system according to claim 15, wherein said 
display device includes a first receiving unit receiving the 

15 first data and a second receiving unit receiving the second data . 

21. A control system for controlling a display device 
including a display unit displaying a motion picture, and a 
pointing unit pointing a position on the motion picture, said 

20 control system comprising: 

a communication unit transmitting the position on the 
motion picture receiving the pointing manipulation to a server 
including a unit recognizing an object in the motion picture 
receiving the pointing manipulation, and receiving, from said 
25 server, information on the object in the motion picture 
recognized by said server; and 

a processing unit executing a predetermined process 
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related to the object recognized. 

22. A control system according to claim 21, wherein said 
display device further includes a receiving unit receiving data , 

5 wherein said receiving unit receives first data containing 

the motion picture, and second data related to the object in 
the motion picture contained in the first data, and 

said processing unit makes said display device output 
the second data related to the object recognized. 

10 

23. A control system according to claim 22, wherein the 
second data contains an image related to the motion picture of 
the first data. 

15 24. A control system according to claim 22, wherein the 

second data is multiplexed with the first data and thus 
distributed . 

25. A control system according to claim 24, wherein the 
20 second data is embedded in a margin of the first data. 

26. A control system according to claim 21, wherein said 
processing unit executes an application program related to the 
object recognized. 

25 

27. A control system according to claim 21, wherein said 
processing unit executes a task related to the obj ect recognized . 
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28. A control system according to claim 21, wherein said 
processing unit commands other task executing device to execute 
a task related to the object recognized via said communication 
unit . 

29. A control system according to claim 21, wherein said 
display device includes a reproducing unit reproducing the motion 
picture recorded on a recording medium, 

said recording medium is recorded with a first motion 
picture and a second image related to an object corresponding 
to the first motion picture, and 

said processing unit makes said reproducing unit reproduce 
from said recording medium the second image related to the obj ect 
recognized and makes said display device display the reproduced 
second image. 

30. A control system according to claim 21, wherein said 
server further includes a unit transmitting an information to 
an add-up device adding up the pieces of information, and 

said processing unit notifies said server of the 
inf Oirmation on the obj ect recognized via said communication unit , 
and makes said add-up device add up results of the recognition 
via said server. 

31. A control system according to claim 21, wherein said 
display device further includes a reproducing unit reproducing 
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the motion picture recorded on recording medium, and 

said communication unit , when the motion picture to be 
reproduced receives the pointing manipulation, transmits, to 
said server, information for specifying this motion picture and 
information for specifying a position where the pointing 
manipulation is effected on the motion picture. 

32 . A server in linkage with a display device including 
a receiving unit receiving data containing a motion picture, 
a display unit displaying the motion picture, and a pointing 
unit pointing a position on the motion picture, said server 
comprising : 

a communication unit receiving information on the position 
where the pointing manipulation is effected from said display 
device ; 

a recognizing unit recognizing the object in the motion 
picture receiving the pointing manipulation on the basis of the 
information received from said display device; and 

a processing unit executing a predetermined process 
related to the object recognized. 

33 . A server according to claim 32 , wherein said processing 
unit transmits the information on the object recognized to said 
display device via said communication unit. 

34 . A server according to claim 32 , wherein said processing 
unit executes a task related to the object recognized. 


- 114 - 


35 . A server according to claim 32 , wherein said processing 
unit , via said communication unit , commands other task executing 
device to execute a task related to the object recognized. 

36. A server according to claim 32, wherein said 
communication unit notifies an add-up server adding up pieces 
of information, of the information on the object recognized, 
and makes said add-up server add up results of the recognition. 

37. A server according to claim 32, wherein said 
communication unit communicates with an E-mail server delivering 
an E-mail, and 

said processing unit, via said communication unit, 
commands said E-mail server to deliver an E-mail related to the 
object recognized. 

38 . A server issuing a command to a data distribution system 
in linkage with a display device including a receiving unit 
receiving data containing a motion picture from a data 
distribution system, adisplayunit displayingthemotionpicture, 
and a pointing unit pointing a position on the motion picture, 
said server comprising: 

a communication unit receiving information on the position 
where the pointing manipulation is effected on the motion picture 
from said display device; 

a recognizing unit recognizing the object in the motion 
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picture receiving the pointing manipulation on the basis of the 
information received from said display device; 

a referring unit referring to an instruction related to 
the object; and 

5 a commanding unit commanding said data distribution system 

to change over the data to be distributed in accordance with 
the instruction. 

39. An add-up device comprising: 
10 a receiving unit receiving pieces of information to be 

added up; 

a category specifying unit retaining category information 
for categorizing the added-up information ; 

an add-up unit categorizing the added-up information in 
15 accordance with the category information and adding up these 
piece of added-up information; 

an information recording unit recording pieces of added-up 
information ruled out of the category information by said add-up 
unit ; and 

20 a generating unit generating a new category for 

categorizing the added-up information recorded on said 

information recording unit, 

wherein said category specifying unit adds the generated 

category to said category information, and 
25 said add-up unit adds up once again the added-up 

information on the basis of the category information with the 

new category . 
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40. An add-up device according to claim 39, wherein the 
added-up information is an answer to a questionnaire. 

5 41. An add-up device according to claim 40, wherein the 

answer to the questionnaire is a piece of information about a 
position on a picture, and 

the category information contains area information 
created corresponding to picture segments . 

10 

42. A processing device comprising: 

a photographing device obtaining an image of a subject; 

a signal generating unit attached to each of a plurality 
of positions on the subject; 
15 a detection unit detecting signals transmitted from said 

signal generating units; 

a recording unit recording a mapping between the image 
and the signal; and 

a processing unit creating, from the signals , information 
20 for defining an occupy area on the image of the subject or a 
part of the subject. 

43 . A server supporting an agency of advertisement for 
a commercial article related to a motion picture segment not 

25 assumed as an object in linkage with a display device including 
a receiving unit receiving a motion picture containing a 
commercial article, adisplay unit displayingthemotionpicture. 
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and a pointing unit pointing a position on the motion picture, 
said server comprising: 

a receiving unit receiving information about the position 
where the pointing manipulation is effected from said display 
5 device ; 

a recognizing unit recognizing an object in the motion 
picture receiving the pointing manipulation on the basis of the 
information received from said display device; 

a recording unit, if unable to recognize the object with 
10 respect to the pointing manipulation, recording data on that 
position; and 

a reporting unit adding up and reporting pieces of 
information on the recorded position. 

15 44 . A server supporting an information research about the 

motion picture in linkage with a display device including a 
receiving unit receiving data containing a motion picture, a 
display unit displaying the motion picture, and a pointing unit 
pointing a position on the motion picture, said server 

20 comprising: 

a receiving unit receiving information about a position 
where the pointing manipulation is effected from said display 
device; 

a recognizing unit recognizing an object in the motion 
25 picture receiving the pointing manipulation on the basis of the 
information received from said display device; 

a recording unit, if unable to recognize the object with 
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respect to the pointing manipulation, recording data on that 
position; and 

a reporting unit adding up and reporting pieces of 
information on the recorded position. 

5 

45. A display device comprising: 

a receiving unit receiving data containing a motion picture 
from a data distribution system; 

a display unit displaying the motion picture; 
10 a pointing unit pointing a position on the motion picture ; 

and 

a communication unit communicating with a server for 
recognizing an object in the motion picture, 

wherein said communication unit, when a motion picture 
15 of first data displayed on said display unit receives the pointing 
manipulation, transmits, to said server, a piece of information 
about a position on the motion picture receiving the pointing 
manipulation and a piece of information for specifying the motion 
picture, 

20 said server commands said data distribution system to 

change over the data to be distributed in accordance with the 
position on the motion picture receiving the pointing 
manipulation , 

said receiving unit receives second data changed over, 

25 and 

said display unit displays the second data. 
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46. A storage medium readable by a machine, tangible 
embodying a program of instructions executable by the machine 
to control a display device including a display unit displaying 
a motion picture and a pointing unit pointing a position on the 

5 motion picture, comprising: 

recognizing an object in the motion picture on which the 
pointing manipulation is effected; and 

executing a predetermined process related to the object 
recognized. 

10 

47. A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said display 
device further includes a receiving unit receiving data, 

said receiving unit receives first data containing the 
15 motion picture, and second data related to the object in the 
motion picture contained in the first data, and 

said predetermined process involves making said display 
device output the second data related to the object recognized. 

20 48 . A storage medium readable by a machine tangible 

embodying a program according to claim 47, wherein the second 
data contains an image related to the motion picture of the first 
data . 

25 49 . A storage medium readable by a machine tangible 

embodying a program according to claim 47, wherein the second 
data is multiplexed with the first data and thus distributed. 
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50 . A storage medium readable by a machine tangible 
embodying a program according to claim 49, wherein the second 
data is embedded in a margin of the first data. 

5 

51. A storage medium readable by a machine tangible 
embodying a program according to claim 47, wherein the motion 
picture of the first data contains an image related to a commercial 
article or a service becoming an advertisement target, and 

10 the second data is an advertisement of the article or the 

service . 

52. A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein the 

15 predetermined process is an execution of an application program 
related to the object recognized. 

53 . A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein the 

20 predetermined process is an execution of a task related to the 
object recognized. 

54 . A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said display 

25 device further includes a communication unit transmitting a 
command to other task executing device, and 

said predetermined process involves commanding, via said 
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communication unit, said other task executing device to execute 
a task related to the object recognized. 

55. A storage medium readable by a machine tangible 

5 embodying a program according to claim 54, wherein said task 
executing device is an E-mail server delivering an E-mail, and 
the command is a request for transmitting E-mail data related 
to the object, and 

said predetermined process involves making the 
10 transmitted E-mail data received via said communication unit. 

56 . A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said display 
device further includes a communication unit transmitting 

15 information to a server adding up pieces of information, and 
said predetermined process involves notifying said server 
of the information related to the recognized object via said 
communication unit, and making said server add up results of 
the recognition. 

20 

57. A storage medium readable by a machine tangible 
embodying a program according to claim 47 , wherein the motion 
picture is formed of a plurality of frames, 

said receiving unit further receives definition 
25 information for defining an area of the object of each frame, 
and 

said recognizing involves collating a position on the 
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motion picture receiving the pointing manipulation with the 
definition information with respect to the frame of the motion 
picture receiving the pointing manipulation. 

58. A storage medium readable by a machine tangible 
embodying a program according to claim 57, further comprising 
a displaying the area in the motion picture with enhancement. 

59 . A storage medium readable by a machine tangible 
embodying a program according to claim 47, further comprising 
receiving information for describing a characteristic of the 
object, and 

said predetermined process involves generating an object 
corresponding to the motion picture on the basis of the motion 
picture and the information for describing the characteristic 
of the object. 

60. A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said 
predetermined process involves, when the pointing manipulation 
occurs during the display of the motion picture contained in 
the first data, commanding said display device to output, after 
an end of displaying the motion picture, the second data related 
to the object recognized by the pointing manipulation. 

61. A storage medium readable by a machine tangible 
embodying a program according to claim 50, wherein said display 
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device further includes a recording unit recording the data, 
and 

said predetermined process involves, when the object is 
recognized from the pointing manipulation during the display 
of the motion picture, commanding said recording unit to record 
the second data related to the obj ect , and commanding said display 
device to output , after the end of displaying the motion picture , 
the second data recorded. 

62 . A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said 
predetermined process involves, when the object is recognized 
from the pointing manipulation during the display of the motion 
picture contained in the first data, commanding said display 
device to effect a first changeover to an output of the second 
data related to the object. 

63. A storage medium readable by a machine tangible 
embodying a program according to claim 62, wherein said 
predetermined process involves commanding said display device 
to executes a second changeover to displaying the motion picture 
after outputting the second data. 

64. A storage medium readable by a machine tangible 
embodying a program according to claim 63, wherein said display 
device further includes a recording unit recording data, and 

said predetermined process involves issuing a command of 
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displaying, after outputting the second data , the motion picture 
from a point of the first changeover recorded on said recording 
unit. 

5 65- A storage medium readable by a machine tangible 

embodying a program according to claim 60, further comprising 
receiving the first data, and receiving the second data. 

66. A storage medium readable by a machine, tangible embodying 
10 a program of instructions executable by the machine including 
a communication unit so as to control a display device including 
a display unit displaying a motion picture and a pointing unit 
pointing a position on the motion picture, to perform method 
steps comprising: 
15 transmitting the position on the motion picture receiving 

the pointing manipulation to a server including a recognizing 
unit recognizing an object in the motion picture receiving the 
pointing manipulation; 

receiving, from said server, information on the object 
20 in the motion picture recognized by said server; and 

executing a predetermined process related to the object 
recognized. 

67 . A storage medium readable by a machine tangible 
25 embodying a program according to claim 66 , wherein said display 
device further includes a receiving unit receiving data, 

said receiving unit receives first data containing the 
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motion picture, and second data related to the object in the 
motion picture contained in the first data, and 

said predetermined process involves making said display 
device output the second data related to the object recognized. 

68. A storage medium readable by a machine tangible 
embodying a program according to claim 67, wherein the second 
data contains an image related to the motion picture of the first 
data . 

69 . A storage medium readable by a machine tangible 
embodying a program according to claim 67 , wherein the second 
data is multiplexed with the first data and thus distributed. 

70. A storage medium readable by a machine tangible 
embodying a program according to claim 69, wherein the second 
data is embedded in a margin of the first data. 

71. A storage medium readable by a machine tangible 
embodying a program according to claim 66, wherein the 
predetermined process is an execution of an application program 
related to the object recognized. 

72. A storage medium readable by a machine tangible 
embodying a program according to claim 66, wherein the 
predetermined process is an execution of a task related to the 
object recognized. 
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73. A storage medixim readable by a machine tangible 
embodying a program according to claim 66 , wherein said 
predetermined process involves commanding, via said 

5 communication unit, other task executing device to execute a 
task related to the object recognized. 

74. A storage medium readable by a machine tangible 
embodying a program according to claim 46, wherein said display 

10 device includes a reproducing unit reproducing the motion picture 

recorded on a recording medium, 

said recording medium is recorded with a first motion 

picture and a second image related to an object corresponding 

to the first motion picture, and 
15 said predetermined process involves reproducing from said 

recording medium the second image related to the object 

recognized and making said display device display the reproduced 

second image. 

20 75 A storage medium readable by a machine tangible 

embodying a program according to claim 66, wherein said server 
further includes a unit transmitting the information to an add-up 
device adding up the pieces of information, and 

said predetermined process involves notifying said server 

25 of the information on the object recognized, and making said 
add-up device add up results of the recognition via said server. 
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76. A storage medium readable by a machine tangible 
embodying a program according to claim 66, further comprising 
reproducing the mot ion picture recorded on said recordingmedium , 

wherein said transmitting involves, when the motion 
5 picture to be reproduced receives the pointing manipulation, 
transmitting, to said server, information for specifying this 
motion picture and information for specifying a position where 
the pointing manipulation is effected on the motion picture. 

10 77. A storage medium readable by a machine, tangible embodying 
a program of instructions executable by the machine including 
a communication unit so as to operate in linkage with a display 
device including a receiving unit receiving data containing a 
motion picture, a display unit displaying the motion picture, 
15 and a pointing unit pointing a position on the motion picture, 
to perform method steps comprising: 

receiving information on the position where the pointing 
manipulation is effected from said display device; 

recognizing the object in the motion picture receiving 
20 the pointing manipulation on the basis of the information 
received from said display device; and 

executing a predetermined process related to the object 
recognized. 

25 78. A storage medium readable by a machine tangible 

embodying a program according to claim 77, wherein said 
predetermined process further includes transmitting an 
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information on the object recognized to said display device. 

79 . A storage medium readable by a machine tangible 
embodying a program according to claim 77, wherein the 
predetermined process is an execution of a task related to the 
object recognized. 

80. A storage medium readable by a machine tangible 
embodying a program according to claim 77, wherein said 
predetermined process involves commanding other task executing 
device to execute a task related to the object recognized via 
said communication unit. 

81. A storage medium readable by a machine tangible 
embodying a program according to claim 77, wherein said 
predetermined process involves notifying an add-up server for 
adding up pieces of information , of the information on the ob j ect 
recognized, and making said add- up server add up results of the 
recognition. 

82. A storage medium readable by a machine tangible 
embodying a program according to claim 77, wherein said 
communication unit communicates with an E-mail server delivering 
an E-mail , and 

said predetermined process involves commanding said 
E-mail server to deliver an E-mail related to the object 
recognized via said communication unit. 
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83. A storage medium readable by a machine, tangible 
embodying a program of instructions executable by the machine 
including a communication unit so as to issue a command to a 
5 data distribution system in linkage with a display device 
including a receiving unit receiving data containing a motion 
picture from said data distribution system, a display unit 
displaying the motion picture and a pointing unit pointing a 
position on the motion picture, to perform method steps 
10 comprising: 

receiving information on the position where the pointing 
manipulation is effected on the motion picture ; 

recognizing the object in the motion picture receiving 
the pointing manipulation on the basis of the information 
15 received from said display device; 

referring to an instruction related to the object; and 
commanding said data distribution system to change over 
the data to be distributed in accordance with the instruction. 

20 84. A storage medium readable by a machine, tangible 

embodying a program of instructions executable by the machine 
including a communication unit and a recording unit , to perform 
method steps comprising: 

receiving pieces of information to be added up; 
25 referring to category information for categorizing the 

added-up information ; 

categorizing the added-up information in accordance with 
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the category information and adding up these piece of added-up 
information ; 

recording pieces of added-up information ruled out of the 
category information in said categorizing; 
5 generating a new set of category information for 

categorizing the added-up information recorded on said 
information recording unit; and 

categorizing and adding up once again the added-up 
information on the basis of the generated category information. 

10 

85 . A storage medium readable by a machine tangible 
embodying a program according to claim 84, wherein the added-up 
information is an answer to a questionnaire. 

15 86. A storage medium readable by a machine tangible 

embodying a program according to claim 85, wherein the answer 
to the questionnaire is a piece of information about a position 
on a picture, and 

the category information contains area information 

20 created corresponding to motion picture segments . 

87. A storage medium readable by a machine tangible 
embodying a program of instructions executable by the machine 
to perform method steps comprising: 
25 obtaining an image of a subject; 

detecting signals transmitted from signal generating 
units attached to a plurality of positions on the subject; 
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recording a mapping between the image and the signal; and 
creating, from the signals, information for defining an 
occupy area on the image of the subject or a part of the subject. 

5 88. A storage medium readable by a machine, tangible 

embodying a program of instructions executable by the machine 
so as to operate in linkage with a display device including a 
receiving unit receiving a motionpictur e containing a commercial 
article, a display unit displaying the motion picture, and a 
10 pointing unit pointing a position on the motion picture, said 
program functioning to support the agency of advertisement for 
the commercial article related to a motion picture segment not 
assumed as an object, to perform method steps comprising: 

receiving information about a position where the pointing 
15 manipulation is effected from said display device; 

recognizing an object in the motion picture receiving the 
pointing manipulation on the basis of the information received 
from said display device; 

recording information on that position if unable to 
20 recognize the obj ect with respect to the pointing manipulation ; 
and 

adding up and reporting pieces of information on the 
recorded position. 

25 89. A storage medium readable by a machine, tangible 

embodying a program of instructions executable by the machine 
so as to operate in linkage with a display device including a 
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receiving unit receiving data containing a motion picture, a 
display unit displaying the motion picture, and a pointing unit 
pointing a position on the motion picture, said program 
functioning to support an information research about the motion 
5 picture, to perform method steps comprising: 

receiving information about a position where the pointing 
manipulation is effected from said display device; 

recognizing an object in the motion picture receiving the 
pointing manipulation on the basis of the information received 
10 from said display device; 

recording information on that position if unable to 
recognize the object with respect to the pointing manipulation ; 
and 

adding up and reporting pieces of information on the 
15 recorded position. 

90. A storage medium readable by a machine, tangible 
embodying a program of instructions executable by the machine 
to control a display device including: 
20 a receiving unit receiving data containing a motion picture 

from a data distribution system; 

a display unit displaying the motion picture; 

a pointing unit pointing a position on the motion picture ; 

and 

25 a communication unit communicating with a server for 

recognizing an object in the motion picture, 
said program comprising: 
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transmitting, when a motion picture of first data displayed 
on said display unit receives the pointing manipulation, a piece 
of information about a position on the motion picture receiving 
the pointing manipulation and a piece of information for 
specifying the motion picture to said server; 

making said server command said data distribution system 
to change over the data to be distributed in accordance with 
the position on the motion picture receiving the pointing 
manipulation ; 

receiving second data changed over; and 

displaying the second data. 

91. A control method of controlling a display device 
including a display unit displaying a motion picture and a 
pointing unit pointing a position on the motion picture, said 
method comprising: 

recognizing an object in the motion picture on which the 
pointing manipulation is effected; and 

executing a predetermined process related to the object 
recognized. 


92. A control method according to claim 91, wherein said 
display device further includes a receiving unit receiving data , 
said control method further comprising: 
25 receiving, via said receiving unit , first data containing 

the motion picture, and second data related to the object in 
the motion picture contained in the first data, and 
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making said display device output the second data related 
to the object recognized. 

93. A control method according to claim 92, wherein the 
5 second data contains an image related to the motion picture of 

the first data. 

94. A control method according to claim 92 wherein the 
second data is multiplexed with the first data and thus 

10 distributed. 

95. A control method according to claim 94, wherein the 
second data is embedded in a margin of the first data. 

15 9 6. A control method according to claim 92, wherein the 

motion picture of the first data contains an image related to 
a commercial article or a service becoming an advertisement 
target, and 

the second data is an advertisement of the article or the 
20 service. 

97. A control method according to claim 91, wherein the 
predetermined process is an execution of an application program 
related to the object recognized. 

25 

98. A control method according to claim 91, wherein the 
predetermined process is an execution of a task related to the 
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object recognized. 

99. A control method according to claim 91, wherein said 
display device further includes a communication unit 
5 transmitting a command to other task executing device, and 

said predetermined process involves commanding said other 
task executing device to execute a task related to the object 
recognized via said communication unit. 

10 100. A control method according to claim 99, wherein said 

task executing device is an E-mail server delivering an E-mail, 
and the command is a request for transmitting E-mail data related 
to the object, and 

said predetermined process involves making the 

15 transmitted E-mail data received via said communication unit. 

101 . A control method according to claim 91 , wherein said 
display device further includes a communication unit 
transmitting information to a add-up server adding up pieces 

20 of information, and 

said predetermined process involves notifying said server 
of the information related to the recognized object via said 
communication unit, and making said server add up results of 
the recognition. 

25 

102. A control method according to claim 92, wherein the 
motion picture is formed of a plurality of frames. 
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said receiving involves further receives definition 
information for defining an area of the object of each frame, 
and 

said recognizing involves collating a position on the 
5 motion picture receiving the pointing manipulation with the 
definition information with respect to the frame of the motion 
picture receiving the pointing manipulation. 

103. A control method according to claim 102, further 
10 comprising displaying the area in the motion picture with 
enhancement . 


104. A control method according to claim 92, further 
comprising receiving information for describing a 

15 characteristic of the object, and 

said predetermined process involves generating an object 
corresponding to the motion picture on the basis of the motion 
picture and the information for describing the characteristic 
of the object. 

20 

105. A control method according to claim 91, wherein said 
predetermined process involves, when the pointing manipulation 
occurs during the display of the motion picture contained in 
the first data, commanding said display device to output, after 

25 an end of displaying the motion picture, the second data related 
to the object recognized by the pointing manipulation. 
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106. A control method according to claim 105, wherein said 
predetermined process further includes recording, when the 
object is recognized from the pointing manipulation during the 
display of the motion picture, the second data related to the 
object, and outputting the second data to said display device 
after the end of displaying the motion picture. 

107 . A control method according to claim 91 , wherein said 
predetermined process involves, when the object is recognized 
from the pointing manipulation during the display of the motion 
picture contained in the first data, commanding said display 
device to effect a first changeover to an output of the second 
data related to the object. 

108 . A control method according to claim 107 , wherein said 
processing step involves commanding said display device to 
executes a second changeover to displaying the motion picture 
after outputting the second data. 

109 . A control method according to claim 108 , wherein said 
display device further includes a recording unit recording data , 
and 

said processing step involves issuing a command of 
displaying the motion picture from a point of the first changeover 
recorded on said recording unit after outputting the second data . 

110. A control method according to claim 105, further 
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comprising receiving the first data, and receiving the second 
data . 

111. A control method of controlling a display device 
5 including a display unit displaying a motion picture and a 
pointing unit pointing a position on the motion picture, said 
method comprising: 

transmitting the position on the motion picture receiving 
the pointing manipulation to a server including a recognizing 
10 unit recognizing an object in the motion picture receiving the 
pointing manipulation ; 

receiving, from said server, information on the object 
in the motion picture recognized by said server; and 

executing a predetermined process related to the object 
15 recognized. 

112 . A control method according to claim 111 , wherein said 
display device further includes a receiving unit receiving data , 

said receiving unit receives first data containing the 
20 motion picture, and second data related to the object in the 
motion picture contained in the first data, and 

said predetermined process involves making said display 
device output the second data related to the object recognized. 

25 113 . A control method according to claim 112 , wherein the 

second data contains an image related to the motion picture of 
the first data. 
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114. A control method according to claim 112, wherein the 
second data is multiplexed with the first data and thus 
distributed . 

5 

115. A control method according to claim 114, wherein the 
second data is embedded in a margin of the first data. 

116 . A control method according to claim 111 , wherein the 
10 predetermined process is an execution of an application program 

related to the object recognized. 

117 . A control method according to claim 111, wherein the 
predetermined process is an execution of a task related to the 

15 object recognized. 

118 . A control method according to claim 111 , wherein said 
predetermined process involves commanding, via said 
communication unit, other task executing device to execute a 

20 task related to the object recognized. 

119. A control method according to claim 91, wherein said 
display device includes a reproducing unit reproducing themotion 
picture recorded on a recording medium, 
25 said recording medium is recorded with a first motion 

picture and a second image related to an object corresponding 
to the first motion picture, and 
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said predetermined process involves reproducing from said 
recording medium the second image related to the object 
recognized and making said display device display the reproduced 
second image. 

5 

120 . A control method according to claim 111 , wherein said 
server further includes a unit transmitting the information to 
an add-up device for adding up the pieces of information, and 

said predetermined process involves notifying said server 
10 of the information on the object recognized, and making said 
add-up device add up results of the recognition via said server. 

121. A control method according to claim 111, further 
comprising reproducing the motion picture recorded on recording 

15 medium, 

wherein said transmitting involves, when the motion 
picture to be reproduced receives the pointing manipulation, 
transmitting, to said server, information for specifying this 
motion picture and information for specifying a position where 
20 the pointing manipulation is effected on the motion picture. 

122. An information processing method for a computer 
including a communication unit to operate in linkage with a 
display device including a receiving unit receiving data 

25 containing a motion picture , a display unit displaying the motion 
picture, and a pointing unit pointing a position on the motion 
picture, said method comprising: 
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receiving information on the position where the pointing 
manipulation is effected from said display device; 

recognizing the object in the motion picture receiving 
the pointing manipulation on the basis of the information 
5 received from said display device; and 

executing a predetermined process related to the object 
recognized. 

123 . An information processing method according to claim 
10 122, wherein said predetermined process further includes 

transmitting an information on the object recognized to said 
display device. 

124. An information processing method according to claim 
15 122 , wherein the predetermined process is an execution of a task 

related to the object recognized. 

125. An information processing method according to claim 
122, wherein said predetermined process involves commanding, 

20 via said communication unit, other task executing device to 
execute the task related to the object recognized. 

126. An information processing method according to claim 
122, wherein said predetermined process involves notifying a 

25 ad-up server adding up pieces of information, of the information 
on the object recognized, and making said add-up server add up 
results of the recognition. 
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127. An informa-tion processing method according to claim 
122 , wherein said communication unit communicates with an E-mail 
server delivering an E-mail, and 

5 said predetermined process involves commanding said 

E-mail server to deliver an E-mail related to the object 
recognized via said communication unit. 

128. An information processing method for a computer 
10 including a communication unit to issue a command to a data 

distribution system in linkage with a display device including 
a receiving unit receiving data containing a motion picture from 
said data distribution system, a display unit displaying the 
motion picture and a pointing unit pointing a position on the 
15 motion picture, said method comprising: 

receiving information on the position where the pointing 
manipulation is effected on the motion pictures- 
recognizing the object in the motion picture receiving 
the pointing manipulation on the basis of the information 
20 received from said display devices- 
referring to an instruction related to the object; and 
commanding said data distribution system to change over 
the data to be distributed in accordance with the instruction. 


25 


129. An information processing method comprising: 
receiving pieces of information to be added up; 
referring to category information for categorizing the 
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added-up information; 

categorizing the added-up information in accordance with 
the category information and adding up these piece of added-up 
information ; 

5 recording pieces of added-up information ruled out of the 

category information in said categorizing; 

generating a new set of category information for 
categorizing the added-up information recorded; and 

categorizing and adding up once again the added-up 
10 information on the basis of the generated category information. 

130. An information processing method according to claim 

129, wherein the added-up information is an answer to a 
guestionnaire . 

15 

131. An information processing method according to claim 

130, wherein the answer to the guestionnaire is a piece of 
information about a position on a picture, and 

the category information contains area information 
20 created corresponding to motion picture segments . 

132. An information processing method comprising: 
obtaining an image of a subject; 

detecting signals transmitted from signal generating 
25 units attached to a plurality of positions on the subject; 

recording a mapping between the image and the signal; and 
creating, from the signals, information for defining an 
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occupy area on the image of the subject or a part of the subject. 

133. An information processing method for operating a 
computer in linkage with a display device including a receiving 

5 unit receiving a motion picture containing a commercial article , 
a display unit displaying the motion picture, and a pointing 
unit pointing a position on the motion picture, said method 
schemed to support an agency of advertisement for the commercial 
article related to a motion picture segment not assumed as an 
10 object, comprising: 

receiving information about the position where the 
pointing manipulation is effected from said display device; 

recognizing an obj ect in the motion picture receiving the 
pointing manipulation on the basis of the information received 
15 from said display device; 

recording data on the position if unable to recognize the 
object with respect to the pointing manipulation; and 

adding up and reporting pieces of information on the 
recorded position. 

20 

134. An information processing method for operating a 
computer in linkage with a display device including a receiving 
unit receiving data containing a motion picture, a display unit 
displaying the motion picture, and a pointing unit pointing a 

25 position on the motion picture, said method schemed to support 
an information research about the motion picture, comprising: 
receiving information about a position where the pointing 
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manipulation is effected from said display device; 

recognizing an object in the motion picture receiving the 
pointing manipulation on the basis of the information received 
from said display device; 
5 recording data on the position if unable to recognize the 

object with respect to the pointing manipulation; and 

adding up and reporting pieces of information on the 
recorded position. 

10 135. An information processing method of controlling a 

display device including: 

a receivingunit receiving data containing a motion picture 
from a data distribution system; 

a display unit displaying the motion picture; 
15 a pointing unit pointing a position on the motion picture ; 

and 

a communication unit communicating with a server for 
recognizing an object in the motion picture, 
said method comprising: 

20 transmitting, when a motion picture of first data displayed 

on said display unit receives the pointing manipulation , a piece 
of information about a position on the motion picture receiving 
the pointing manipulation and a piece of information for 
specifying the motion picture to said server; 

25 making said server command said data distribution system 

to change over the data to be distributed in accordance with 
the position on the motion picture receiving the pointing 
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manipulation ; 

receiving second data changed over; and 
displaying the second data. 


